1. Field of the Invention
The invention relates to virtual machines associated with a computer system and movement of those virtual machines between computer systems separated by a wide area network.
2. Description of the Related Art
Businesses are expected to migrate, consolidate and maintain their data centers without any downtime while providing high availability for applications.
Typical data center organization is shown in FIG. 1. Two data centers 100, 150 are illustrated. Each has a series of application server clusters 102, 152 which execute the actual applications, such as in a SaaS (software as a Service) architecture. Data is stored in a storage fabric 104, 154. Access to the application server clusters 102, 152 is shown as being through web server clusters 106, 156, though more direct access at the local area network (LAN) layer is common. A site load balancer 108, 158 distributes incoming requests across the web servers in the web server clusters 106, 156. A global load balancer 110 is connected to the Internet 112 to balance load between the data centers 100, 150.
VMware's vMotion enables the movement of a live virtual machine (VM) from one server to another without any downtime of the server or the application or impact on the end user. During vMotion, the active memory and an accurate execution state of a virtual machine is transmitted over a high speed network from one physical server to another. Prior to performing the migration, the vMotion application establishes a network connection between the source and the destination servers and various resources like CPU, memory, storage, and network connections are checked for compatibility. In the initial stages of migration, the inactive pages of the VM state are moved, leaving behind a small working set of active pages. VM configuration and the device information are copied to the destination server and worker processes are created. The VM memory is copied into the destination while the server is still running. Memory writes are intercepted and are used to track actions that occur during the migration. After most of the VM's state has been moved, the VM is then paused and transferred to the destination host. The access to the storage is moved from one host to another and the VM is reset in the destination. Two Transmission Control Protocol (TCP) sessions are established during vMotion, as shown in FIG. 2. While one is used for migration of the virtual machine, the other is used to maintain the VM state consistency among the two servers while the VM is in transit.
vMotion imposes strict requirements on the network. These requirements can be broken into:
1) High Bandwidth: VM migration requires a minimum of 1 GB bandwidth.
2) Encryption: VM migration data is usually not encrypted and hence there is a need to support encryption to protect the data.
3) Storage Accessibility: Along with VM migration, the shared storage associated with VM may need to be migrated to allow quicker access.
4) Latency: The migration can only work if the latency is less than 5 ms and low packet loss and packet duplication and other impairments.
5) Application Continuity: There should not be any impact on the end user while the migration is in progress.
There is a need for a technique that can reduce the latency incurred in migrating live services, applications and virtual machines from one data center to another across a wide area network (WAN) infrastructure. This challenge gets more complicated when the migration needs to be done with zero impact on the user and it becomes virtually impossible if the migration has to be performed over long distances. Some of the challenges include overcoming the WAN latency observed by the applications while making efficient use of available bandwidth and also devising a mechanism to accelerate the application migration.
TCP is primarily used as a transport layer protocol for most of the migration applications. Round trip times (RTTs) observed will be very high because of the high end to end network latency. Huge RTT values influence the congestion control mechanism and cause a very long slow startup phase and hence also result in sub optimal bandwidth usage. If a congestion event occurs in such a situation, the connection window size will be dropped to zero and the slow start phase is triggered all over again, causing further reduction in the throughput. Various Transmission Control Protocol/Internet Protocol (TCP/IP) characteristics that contribute toward inefficient use of the available bandwidth and reduction in overall throughput are:
1) Window Size: The TCP window size is a 16 bit field advertised in the TCP header which limits the window size to a maximum value of 64 KB. TCP can only send 64 KB before it receives acknowledgement. Since 64 KB takes 0.5 ms to transmit across a 1 Gbps link, even 1 ms latency could cut the TCP performance by a factor four. To maintain the line rate at 1 Gbps with 1 ms of network latency, a window size of 128 KB is needed. Higher network latencies demand bigger window sizes to maintain the same line rate. For example, at 1 Gbps network speed and 60 ms RTT, a receive window size of 8 MB is needed. The application servers cannot handle such high demand on memory since they handle multiple TCP connections. A realistic window size limit is 3 MB, as shown in FIG. 3.
2) Network Reordering: Dynamically load balancing networks cause excessive false congestion events due to reordering done at the receiver. Due to the dynamic load balancing nature of the network, the packets might arrive out of order with some packets undergoing more delay to reach the receiver than the others. If the receiver performs reordering without waiting for the packets to arrive, it will generate duplicate acknowledgements. This triggers a fast retransmission operation and hence results in unnecessary duplicate packet retransmissions by the sender. A false congestion event is believed to occur and TCP enters the fast recovery stage. The connection window is reduced by half, which brings down the throughput.
3) Retransmission Timer timeouts: Packet loss has a detrimental effect on TCP's performance. If fast retransmission is not triggered, it is required to wait for 500 ms (usually) to act on the congestion in the network, which slows down the recovery and decreases the throughput. The timeout value increases with every timeout, which further slows operations.
4) Slow start and congestion avoidance: Once a timeout event occurs, the connection window (cwnd) is reduced to zero. Now, the rate at which the packets are injected into the network depends on the rate of the acknowledgements received. Although this seems like an exponential increase in the connection window, the growth becomes linear when TCP enters the congestion avoidance stage. The slow start and the congestion avoidance in conjunction with the high retransmission timeout values, slows down the recovery of the TCP from a congestion event, which thoroughly impacts the overall throughput. See FIGS. 4 and 5 for examples of congestion avoidance operations.
An example of vMotion is provided in FIGS. 6A-6O. Two data centers 600, 650 are illustrated having identical configurations. A virtual machine server 602, 652 is connected to a network 604, 654, typically a LAN, with additional servers 606, 656, 608, 658 attached. A storage unit 610, 660 is also attached to the network 604, 654. A router 612, 662 is connected to the network 604, 654 to allow interconnection of the data centers 600, 650 using a Virtual Private LAN Service (VPLS) link. VM server 602 has two VMs, VM1 614 and VM2 616, and data, Data1 618, Data2 620, Data3 622 and Data4 624. For this example all four data blocks 618-624 are associated with VM1 614. In the example VM1 and Data1-Data4 are migrated from VM server 602 to VM server 652. Initially the VM1 614 and Data1 618 are operated on. VM1 614 is replicated and Data1 618 is moved. VM server 652 sends a response (RSP) message 626 back to VM server 602 to indicate the success of that portion of the move. In response, VM server 602 sends Data2 to VM server 652. Similar operations occur to move Data3 622 to VM server 652. VM server 602 receives the related RSP message 628 for Data3 622. Data4 624 is moved to VM server 652 which generates an RSP message 630 back to VM server 602. As this is the last of the data, VM server 602 provides a COMMIT message 632 to VM server 652. VM server 652 replies with a DONE message 634. When VM server 602 receives the DONE message 634, it removes the VM1 614, which results in the VM1 614 being fully migrated or moved.
If the application was migrated and the associated storage is not migrated, the disk/tape access will get very costly as they have to go over the WAN infrastructure for every read/write operation. The storage migration, if performed, will also be affected by the WAN latencies as mentioned above.